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ABSTRACT 



In response to the increased use of the Internet for 
information access and dissemination, and the explosion in the availability 
of resources which may be of interest to end-users, a number of services are 
now available which provide access to selected and evaluated resources. Many 
existing Internet search tools are inadequate due to the high rate of recall, 
low precision, and a lack of filtering. Consequently, the development of 
evaluative tools is potentially useful for both end-users and information 
professionals. This paper discusses and compares a number of sites which seek 
to select, evaluate, review, or describe networked information sources. The 
paper looks at an overview of the sites' services, resource selection, level 
of description, subject classification and organization, and review criteria 
used. The paper is the result of work carried out by the Organizing Medical 
Networked Information (OMNI) Advisory Group for Evaluation Criteria which has 
been looking at a range of services available via the Internet for accessing 
selected and evaluated medical information sources, and the criteria they 
use. Internet review services differ greatly in terms of coverage, formality, 
and intended audience- -sites tend to be aimed either at a general audience, 
covering a broad range of subject areas and produced informally, or 
specifically at academic and research users and involving the use of formal 
evaluation criteria, implemented by information professionals or subject 
experts. URLs (uniform resource locators- -Internet addresses) of Internet 
review services are appended. (Contains 13 references.) (Author/SWC) 
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Abstract: In response to the increased use of the Internet for information access and dissemination, and 
the explosion in the availability of resources which may be of interest to end-users, a number of services 
are now available which provide access to selected and evaluated resources. Many existing Internet 
search tools are inadequate due to the high rate of recall, low precision and a lack of filtering. 
Consequently, the development of evaluative tools is potentially useful for both end-users and infor- 
mation professionals. 

This paper will discuss and compare a number of sites which seek to select, evaluate, review or describe 
networked information sources. The paper is the result of work carried out by the OMNI Advisory Group 
for Evaluation Criteria which has been looking at a range of sen/ices available via the Internet for 
accessing selected and evaluated medical information sources, and the criteria they use. 

Keywords: Internet, review services, information evaluation 



1 . Introduction 

The Internet initially developed as a global communications channel, allowing the world-wide exchange of 
messages via e-mail. With the success of the World Wide Web and the widespread adoption of browsers such 
as Mosaic and Netscape Navigator, there has been an unprecedented explosion in the number of users of 
networked information as well as in the number of information providers: 

‘The Internet is frequently described by its users ... as a “realm of pure mind”. Anything which is thinkable can be 
expressed to an audience of hundreds of thousands’ (Ref 1). 

Commercial organisations now recognise its potential for global advertising and selling. Individual organisa- 
tions, including universities and other research institutions world-wide, rely on the Internet as a mechanism for 
disseminating promotional material, course details, research profiles and so on. A number of existing journals, 
magazines and other publications produce parallel online equivalents and publishers have developed e-zines and 
online journals where the definitive version is electronic. The media generally has begun to move into the inter- 
active age with online programme information and contact details. In addition the personal home page, listing 
such materials as individuals’ hobbies and interests, pictures of pets and links to pop stars’ pages, is a continuing 
phenomenon of the networked environment. Indeed, the list of available resources is endless. 

However, ease of access and dissemination has resulted in a number of problems for network users and 
recent critics have begun to consider the problems associated with the availability of so much information. Davey 
Winder suggests that threats to the Internet not only include censorship, pornography and commercialisation but 
also information overload: ‘there’s now so much information available on the Net, and so many people trying to 
access it, that something, somewhere, has got to burst.’ Information overload presents two central problems: 
firstly, ‘finding the information you want in a sea of dross’, and secondly ‘getting connected and managing to 
download the information once you’ve located it’ (Ref 2). Likewise, Louis B. Rosenfeld identifies similar problems 
which relate to finding information — determining the existence and location of resources, using the different 
software available, as well as problems of ‘scale’ and ‘quality’: 

‘Scale: ... the user may win the battle of the Internet, namely, finding a useful information resource, only to lose 
the battle of traditional information retrieval, namely being overwhelmed by too many hits from a large database.’ 
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‘Quality: The majority of the information found on the Internet is lacking in terms of currency, authority and 
comprehensiveness; in addition, only a handful of resources is refereed in any way. Unfortunately there are few 
trusted mechanisms for assessing the quality of the Internet’s information resources’ (Ref 3). 

Matthew Ciolek has expressed his concern about the growth of what he calls ‘multi-media mediocrity’ in an 
article in which he posits that the ratio of total volume of networked information (measured in megabytes) to infor- 
mation useful to scholars may be much smaller than imagined. He is also concerned with long-term trends and 
prospects for the quality and reliability of WWW-based information resources (Ref 4). This concern is expressed 
in part through the creation of the Information Quality WWW Virtual Library, a set of pages which keeps track of 
online resources relevant for evaluation, development and administration of high quality factual/scholarly 
networked information systems (Ref 5). 

Much research work has been concerned with the development of (WWW-based) search and retrieval tools 
with the aim of providing effective subject access to resources. Examples include automatically generated 
indexes such as Lycos and Alta Vista, and manually created ‘virtual libraries’ or catalogues, such as Yahoo!. 
Existing search and retrieval tools may assist in the identification of information sources but as the number of 
resources available via the Internet continues to grow, so the number of resources retrieved becomes increas- 
ingly unmanageable. Often a search results in the user being faced with an unwieldy quantity of pointers or links, 
with no clear way of restricting search results. The user must expend time and energy sifting through results, only 
to find that material may be out-dated, no longer available, or that a number of individual pointers link to different 
parts of the same source. Although some search engines offer sophisticated relevance ranking mechanisms and 
these may improve recall, the first items returned by the search tools are not necessarily the most relevant to the 
user, and casual users without a background in information retrieval are often not prepared to invest time in 
learning to use the ranking mechanisms effectively. Furthermore, such practices as word spamming may serve 
to undermine the possible benefits of relevance ranking. 

Prior to the availability of networked information, users could have some confidence in the quality of infor- 
mation available to them. For example, publishers of popular material have a commercial imperative to produce 
‘quality’ material while academic publishers seek to maintain quality through the editing and refereeing 
processes. However, the Internet enables anyone to become an author and provider of information authors 
need not adhere to concerns for an audience but may publish at will. Material may be purely egocentric or of an 
ephemeral nature and of little interest to others. In particular there is considerably more vanity publishing in the 
networked environment (for example the personal home page), which has been noted by Hope Tillman (Ref 6). 
Thus individual users are now faced with the need to identify not only relevant material but also quality material. 

In response to such problems, a number of sources and services have been developed which attempt to 
select, evaluate and review networked information. Journals such as Online , Database and the Library Journal 
regularly print reviews of Internet resources and College and Research Libraries News features an Internet reviews 
section in each issue. Ariadne (Ref 7), a two-year parallel publishing journal project funded under the UK’s 
Electronic Libraries Programme, also includes regular reviews. Such sources may assist end-users but are 
primarily aimed at the information professional audience. 

On the Internet itself a number of reviewing services have been developed. Such services seek to select, 
evaluate or review networked sources in order both to identify relevant materials and also assess the quality of 
the material retrieved. The development of such evaluative tools is potentially useful for both end-users and infor- 
mation professionals alike. However, such services are themselves of varying usefulness and quality. This paper 
seeks to compare a number of such sites, offering a guide to the types and variety of services which are now 
available. 



2. Internet review sites 

A number of review services were selected for comparison: A#1 Quality Directory, Cyberstacks, Excite, 
Healthweb, InfoFilter, Magellan (McKinley), Medical Matrix, OCLC NetFirst, Organising Medical Networked 
Information, Point Web Reviews, Science and Engineering Network News, Six Senses and WebCrawler Select 
(see Appendix for URLs). Originally these services were considered with specific, qualitative questions in mind: 

• does the site clearly state the criteria it uses for the selection and inclusion of resources? 

• is the evaluation site selective in the type of resource included? 

• is the evaluation site selective in the subject area it covers? 

• if resources are given a rating (e.g. stars or points), are the criteria for the ratings stated? 

• do resource descriptions/evaluations include the name of the evaluator ? 

• is there any indication of the expertise/ profession of the evaluator ? 

• do resource descriptions/evaluations include the date on which the evaluation was written? 

• is there a mechanism for the regular updating of reviews? 

• are the URLs for resources included in the database regularly checked for changes, either automatically by 
link-checking software or by a human? 
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• are ‘dead’ links identified and removed promptly? 

• does the site include a search facility ? 

However, not all sites explicitly provide details about such aspects of their service. Where information is 
provided it can often be found under the services’ FAQ or the ‘about’ link. For example, both Magellan and Excite 
include extensive FAQs about their service, staff and criteria for including resources. 

In the event, it was possible to draw together a number of common traits into four broad strands: resource 
selection; the level of description; subject classification and organisation; and the evaluation criteria used. 

2.1 . Overview of the services 

The services selected for comparison vary greatly in terms of the types of resources covered, the subject 
coverage and the processes of resource selection, description, organisation and evaluation. 

Generally the sites may be divided into three groups, noted in Table 1 overleaf. 

The first group includes A#1, Excite, Magellan, Point and WebCrawler Select. These are all fairly informal 
services which have no particular subject focus. No intended audience is explicitly stated or apparent and the 
guidelines on resource selection and evaluation criteria (where available) tend to be vague and informal. Such 
services are not produced by those professionally involved in providing access to information, but rather network 
enthusiasts and ‘cyberwriters’. Point, for example, is produced by ‘a mix of long time Web fanatics, some profes- 
sional enthusiasts and even a few students.’ 

The second group, comprising CyberStacks, InfoFilter, OCLC NetFirst and SENN, is also broad in terms of 
subject coverage but tends to be more formal. Either information professionals or subject experts are involved in 
resource selection, description and evaluation; the services are aimed towards the academic or ‘serious’ infor- 
mation user; and more detailed selection and evaluation guidelines are available. 

The last group of four services are those with a subject focus concentrating specifically on health sciences, 
medical and biomedical sources: HealthWeb, Medical Matrix, Organising Medical Networked Information (OMNI) 
and Six Senses. With the exception of Six Senses these services take a more formal approach, involving infor- 
mation professionals and subject experts in the provision of services for academic and research users. Formal 
criteria and guidelines on selection and description are available. Although Six Senses is concerned with a similar 
subject area it follows a hybrid approach. Even though the review panel includes a registered nurse and a 
practising clinician, the emphasis of the evaluation criteria tends to coincide more with the services listed in the 
first group and the rating scheme produces results that can sometimes be difficult to comprehend. 

An indicative summary of a range of features offered by the services can be seen in Table 1. 

2.2. Resource selection 

The review services examined vary in the approach and thoroughness with which new sources are identified and 
selected. The informal sites, such as Point, are vague in describing the process of resource selection. Point 
claims that excellence is the only criterion for inclusion and that it ‘makes no distinction between commercial, 
private, or student pages.’ 

Alternatively, sites such as HealthWeb and OMNI are more specific. These sites primarily involve health 
sciences librarians and information professionals in the selection and evaluation of sites. Both state that their 
interest is in providing access to critically-evaluated resources rather than an exhaustive index of health-related 
information. Medical Matrix seeks to be comprehensive in its coverage of ‘global resources that can assist in 
patient care’, but shares a similar concern for quality resources. 

The majority of services invite users to recommend sites, although some indicate that these will only be 
treated as suggestions. For example, Magellan does not guarantee that a suggested resource will be reviewed, 
only that it will be added to its searchable database. Submissions are subject to an editorial review process and 
sites are only accepted on the basis of editorial discretion. GNN Select typically adds no more than 1 % of recom- 
mended sites. None of the sites automatically include reviews submitted by casual browsers. 
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Group 3 

HealthWeb 1223 yes no yes 

Medical Matrix 2410 yes no yes 

OMNI 800+ yes no yes 

Six Senses 1 00+ yes yes yes 



2.3. Level of description 

The level of description offered also varies according to the nature of the service. For example, InfoFilter provides 
extensive reviews of resources. The reviews are analytical and provide clear guidance on whether a resource is 
likely to be of interest to a target audience. Other services offer reviews which are brief and merely indicative of 
the site described while some are simply annotations or descriptions, with little evaluative detail. Only 
CyberStacks, InfoFilter and Six Senses include the name of the reviewer. 

A number of sites employ a numerical rating system. For example, Point rates resources between 0 and 50 
for content, presentation and experience while Magellan rates sites by awarding up to 10 points for depth, ease 
of exploration and ‘Net appeal’. 

Disappointingly only two services (InfoFilter and Six Senses) explicitly include the date of the review, although 
an approximate date can be determined in some cases (for example, if the site review is included in a ‘what’s 
new’ section). Point can be searched in order of review date but this information is not included in the site 
annotation. In addition to including the review date, the rapidly-changing nature of the Internet requires regular 
checking of sites to verify links and to ensure reviews are up-to-date. Six Senses claims to re-evaluate sites every 
two to three months. However, on closer examination some reviews had been written over six months previously 
and did not appear to have been updated. Medical Matrix states ‘entries are continuously updated and verified’. 

2.4. Subject classification and organisation 

Many services are browsable by subject category. NetFirst offers both Library of Congress Subject Headings and 
Dewey Decimal Classification numbers. The CyberStacks project has begun to list Internet review services and 
subject collections by their own classification scheme (Ref 8). Perhaps surprisingly for a review service produced 
by librarians, InfoFilter has chosen not to include classification codes. NetFirst offers both Library of Congress 
Subject Headings and Dewey Decimal Classification numbers. Of particular interest to medicine are sites which 
organise resources by the National Library of Medicine classification codes (NLM) and Medical Subject Headings 
(MeSH), and here OMNI is a good example. 

While most general review sites (such as Excite, Magellan and Point) tend to include a section of 
‘health/medicine’ resources, they do not offer such formalised organisation as the services specialising in 
biomedical resources. 

The majority of review sites are also searchable by keyword. Where a controlled vocabulary is included, such 
as the use of MeSH, consistency is improved. However, the use of thesauri is not widespread. 

2.5. Review criteria used 

The nature of the criteria used to evaluate and review information sources is dependent upon the providers of the 
service. Librarians and other information professionals have long been involved in the critical selection and evalu- 
ation of information sources. Evaluation criteria have been developed over the years for the evaluation of 
reference works, monographs, audiovisual materials and more recently online and CD-ROM sources. 
Consequently, those services provided by information professionals tend to use fairly formal criteria. For 
example, CyberStacks bases its evaluations on criteria taken from the American Library Association manual for 
Reference Collection Development. Resources are evaluated under the broad areas of authority, accuracy, 
clarity, uniqueness, recency/timeliness, community needs and whether any other favourable reviews are 
available. 

As part of the InfoFilter project, participants are actually concerned with criteria development and a similar set of 
criteria has been drawn up. Boyd R. Collins, founder of the InfoFilter project states: 

‘The current reviewing mechanisms we have for print materials arose from the need for objective standards in the 
face of the mass of materials that are produced yearly. The idea behind the InfoFilter project is to leverage the 
same well-developed and tested principles of reference book reviewing to the sources available on the Web’ (Ref 
9). 

Likewise, the subject-oriented services use criteria based on those traditionally used by information profes- 
sionals for evaluating other sources of information. 

The ‘popular’ sites, such as the A#1 Quality Directory, Point and Magellan tend to be concerned with access 
and presentation issues. Content issues are a consideration (Point considers, ‘Just how broad, deep, and 
amazingly thorough is the information? Are there good links? Good clips? Is it accurate? Complete? Up-to- 
date?’). However, criteria used in the A#1 Directory include the time taken for a page to load (the service aims to 
provide links to resources which load quickly using an ‘average’ computer), pricing, page design and layout and 
ease of navigation. Likewise, Point asks: 

‘Is the page beautiful? Colourful? Easy to use? Does it lead visitors through the information nicely? Does it use 
video, audio and original graphics? Does it break new ground? Is it fun? Is it worth the time? Will we recommend 
it to friends? All things considered, does this site deliver the goods?’ 
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Magellan: 



‘Is it well-organised and easy to navigate? Is it innovative? Does it appeal to the eye or the ear? Is it funny? Is it 
hot, hip, or cool? Is it thought-provoking? Does it offer new technology or a new way of using technology?’ 



3. User-centred review services: the key to successful 
evaluation? 

The analysis of key attributes of a range of review services highlights a number of issues that bear on the actual 
utility of such sites. A clear basis for resource selection, usually on behalf of a stated target audience, often 
underlies the development of services that have applied formalised standards of resource evaluation, description 
and organisation. 

For example, for the casual user the existence of classification codes or subject headings may be of 
peripheral interest only; but for the searcher who is familiar with the vocabulary structure of their own subject area 
(e.g. medicine), the availability of a consistent means of accessing this is welcome, such as through the use of a 
controlled vocabulary. 

However it is possibly in the criteria for evaluation that the distinctions among services are most clearly visible. 
In most subject-based review services, simple inclusion of resources usually acts as an indicator of quality. 
In contrast, in the absence of a clear target audience, a number of general review sites (such as Point and 
Magellan) assign ratings and scores, alluding to objectivity. Yet, more often than not, no special weighting is 
employed and the figures are simply allocated or uniformly aggregated to provide a magical final score or star 
rating. This leads to an interesting distinction compared to the subject-oriented evaluative initiatives. Medical 
Matrix, for example, gives explicit weighting to the evaluation criteria used, which are far greater for issues of 
scope and content than for presentation and innovation. While the display of ratings and scores and the use of 
presentation and access criteria may offer the end-user some indication of the nature of a resource, it is 
questionable how useful such an evaluation can be. 

Norman Stevens, writing in 1986, lists 18 criteria that could be considered when evaluating reference books. 
Yet he also notes: 

‘Not all of these elements carry equal weight in respect to a single work, and, in some cases, a number of them 
may not apply at all. Judgement in determining what elements do apply and what weight is to then be given to 
them is especially critical ... A number of standards used to evaluate a reference book may be of little significance 
to a user’ (Ref 1 1). 



Richard Hopkins further elaborates: 

‘Any system should be based on the actual information needs of the users involved, not just on available 
technology or on the subject of the area of study under consideration (expressed in a different way the system 
must be “user-driven” rather than “technology-driven” or “content-driven”) ... Users bring identifiable criteria or 
priorities to information systems; that is they have certain expectations that they want the information system to 
fulfil* (Ref 12). 



Ease of use, noise reduction, quality, adaptability, time saving and cost saving are examples of these. 

In a recent review of issues relating to database information quality, Chris Armstrong reported on surveys of 
information professionals which were conducted independently by EUSIDIC (the European Association of 
Information Services) and CIQM (the Centre for Information Quality Management). Both these organisations 
which invited respondents to rank their views of ten evaluation criteria. The top five rankings were (Ref 12): 



EUSIDIC 

Coverage 

Accessibility 

Timeliness 

Consistency 

Accuracy 



CIQM 

Accuracy 

Consistency 

Timeliness 

Accessibility 

Coverage 



The criteria used in the surveys were originally developed during the Southern California Online User Group 
(SCOUG) 1990 Annual Retreat and are now widely used in the evaluation of online and CD-ROM information 
sources (Ref 13). The consistently higher ranking of the five criteria above (compared to the remaining five: value, 
documentation, harmonisation, output and support) might indicate their suitability for weighting any evaluation. 
While the SCOUG criteria were originally developed and subsequently ranked by information professionals, it is 
interesting to note how broadly similar they are to end-users’ perceptions. 

Alison Cooke is currently conducting research into how end-users of the Internet select and evaluate 
networked information sources. Preliminary results indicate that users are principally concerned with aspects 
such as the authority, accuracy and currency of information. Information presentation affects some users but the 



ERIC 



Online Information 

Page 38 



96 Proceedings 



8 



t 



overriding factor is information content. The limited attention that most commercial services pay to content evalu- 
ation would not realistically offer the user much guidance on such aspects as the authority, factual accuracy, 
reliability or currency of the source, all of which are central issues to consider when evaluating the quality of infor- 
mation. 



4. Conclusion 

The development of services which seek to select, evaluate or review networked information sources has been 
a growth area during 1995 and 1996, and a number of sites are now available. Such services differ greatly in 
terms of coverage, formality and intended audience — sites tend to be aimed either at a general audience, 
covering a broad range of subject areas and produced informally, or specifically at academic and research users 
and involving the use of formal evaluation criteria, implemented by information professionals or subject experts. 
Some services have elected to take a specific subject focus. 

The quality of the reviews provided is affected by factors such as the providers of the service and the size of 
the site/number of resources evaluated. Small sites provided by library and information professionals, such as 
OMNI, offer detailed evaluations of resources but are more restricted in the number of sites they can cover. 
Alternatively, the more general and ‘popular’ services have a broader coverage and are much larger and of 
interest to the non-specialist, but provide less evaluative detail. In addition the criteria used by such general sites 
are far removed from the criteria with which information professionals would evaluate other resources. 

At present choosing from amongst the evaluation sites seems to be a variant of the precision vs. recall 
dilemma: such services can either produce a limited number of quality reviews or a large number of reviews 
resulting from much less rigorous criteria. As discussed previously, the broader, informal sites tend to focus on 
aspects such as the layout and presentation of sources in order to assess the ‘quality’ of different information 
sources, and pay little attention to content. However, such criteria may not only provide limited detail on the 
quality of sources but may also be of little value to potential users. Thus it is possible to differentiate between the 
sites in terms of how meaningful the evaluations really are. 

The authors of this paper concur with Richard L. Hopkins (cited earlier) that the major problems in future 
information systems will revolve around the processes of reducing the amount and raising the quality of infor- 
mation brought to the attention of the user.’ Evaluative sites with little subject focus, or which do not make formal 
guidelines available, may cover more resources but ultimately users are likely to be concerned with aspects such 
as information accuracy and reliability, and thus require a detailed and meaningful review of the quality of different 
sources. 
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Appendix: URLs of Internet review services 



A#1 Quality Directory 

Cyberstacks 

Excite 

Healthweb 

InfoFilter 

Magellan (McKinley) 
Medical Matrix 
OCLC NetFirst 
OMNI 
Point 

Science & Engineering 
Network News 
Six Senses 
WebCrawler Select 



http://www.illuminatus.com/clients/a1source/ 

http://www.public.iastate.edu/-CYBERSTACKS/ 

http://www.excite.com/Subject/ 

http://hsinfo.ghsl.nwu.edu/healthweb/ 

http://www.usc.edu/users/help/flick/lnfofilter/ 

http://www.mckinley.com/ 

http://www.slackinc.com/matrix/ 

http://www.oclc.org/oclc/netfirst/9352nf/netflyer.htm 

http://www.omni.ac.uk/ 

http://www.pointcom.com/ 

http://www.senn.com/ 

http://www.sixsenses.com/ 

http://webcrawler.com/select/ 
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